Image processing apparatus, program, and method for performing preprocessing for movie reproduction of still images

ABSTRACT

An apparatus takes in a plurality of still image frames, to generate an interpolated image frame. Next, moving addition of the still image frame and an interpolated image frame is performed in time-axis direction, to generate a superimposed image frame. At this time, high spatial frequency components in the superimposed image frame are suppressed strongly as alienation of motion between the still image frames becomes larger. Movie is reproduced by sandwiching these superimposed image frames between the still image frames. As a result of this, the movie reproduction with smooth motion is made possible even from a group of the still images photographed by an electronics still camera and the like.

CROSS REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2004-276063, filed on Sep. 22, 2004, theentire contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing technique thatdisplays a plurality of still images as smooth movie.

2. Description of the Related Art

A conventionally known technique sequentially reproduces frame-by-framea plurality of still image frames which are photographed by anelectronic camera, with the operation of a frame step button by the user(for example, Japanese Unexamined Patent Application Publication No. Hei9-116852 and the like).

Incidentally, the present inventor has considered realizing fine moviereproduction by using the plural still image frames which arecontinuously photographed as his or her material. When such a functionis realized, images stored as the still images can be disclosed asmovies in electronic albums or websites on the Internet, which makesimage appreciation even more enjoyable.

Moreover, when the plural still image frames are reproduced as a movieimage by a image editing software or the like, it is possible to view avast number of still images in a short period of time. Additionally, itis also possible to quickly find a desired still image from the vastnumber of the still images by this movie reproduction.

In such a movie reproduction, approximately 18 to 30 image frames persecond are generally necessary in order to display smooth motion like infilms or television programs.

Meanwhile, when the above-described electronic camera performedstill-image photographing, its photographing intervals wereapproximately 2 to 8 frames per second, even with a continuousphotographing mode whose speed is relatively high. When the still imageshaving such photographing intervals were simply reproducedframe-by-frame, motion of the images became awkward and hence fine moviereproduction was difficult.

Further, still images are often photographed by controlling imageblurring so as to be suitable for still image appreciation. When thestill images with less image blurring are simply reproducedframe-by-frame, a linkage of motion is hardly established between theimages and movie reproduction becomes awkward.

SUMMARY OF THE INVENTION

Therefore, it is an object of the present invention to provide an imageprocessing technique that realizes fine movie reproduction by using, asmaterials, still images with long photographing intervals and the stillimages with less blurring which are suitable for appreciation as thestill images.

Hereinafter, an explanation on the present invention will be given.

(1) An image processing apparatus according to the present inventionexecutes preprocessing to smoothly perform a movie display of image datacomposed of a plurality of still image frames obtained by still-imagephotographing, including an image inputting unit, a frame interpolatingunit, a superimposing unit, and a smoothing unit.

The image inputting unit imports the plurality of the still imageframes.

The frame interpolating unit applies frame interpolation to theplurality of the still image frames in time-axis direction in order ofphotographing, to generate an interpolated image frame.

The superimposing unit performs moving addition of either one of “thestill image frame and the interpolated image frame” and “a plurality ofthe interpolated image frames with each other” in the time-axisdirection, to generate a superimposed image frame.

The smoothing unit increases smoothness of the movie display bysuppressing more high spatial frequency components in the superimposedimage frame as alienation of motion between the still image framesbecomes larger.

(2) Preferably, the smoothing unit increases the smoothness of the moviedisplay by suppressing more of the high spatial frequency components inthe superimposed image frame as a charge storage time at the time ofphotographing the still image frame becomes shorter.

(3) Preferably, the smoothing unit increases the smoothness of the moviedisplay by suppressing more of the high spatial frequency components inthe superimposed image frame as a photographing interval of the stillimage frame becomes longer.

(4) Preferably, the smoothing unit increases the smoothness of the moviedisplay by suppressing more of the high spatial frequency components inthe superimposed image frame as a moving amount (motion amount) of theimage between the still image frames becomes larger.

(5) Preferably, the smoothing unit suppresses more of the high spatialfrequency components in the superimposed image frame as a maximum motionamount among the motion found from a plurality of places in an imagearea becomes larger.

(6) Preferably, the smoothing unit increases the smoothness of the moviedisplay by increasing a number of the frames of the interpolated imageframes which the frame interpolating unit generated, and by suppressingthe high spatial frequency components in the superimposed image frame.

(7) Preferably, the smoothing unit increases the smoothness of the moviedisplay by increasing a number of frames to be performed movingaddition, and by suppressing the high spatial frequency components inthe superimposed image frame.

(8) Preferably, the smoothing unit increases the smoothness of the moviedisplay by commonly using either one of the still image frame and theinterpolated image frame when generating the superimposed image frameswhich are temporally adjacent to each other.

(9) Preferably, the smoothing unit increases the smoothness of the moviedisplay by increasing a number of either one of the still image frameand the interpolated image frame to be used commonly, and by suppressingthe high spatial frequency components in the superimposed image frame,when generating the superimposed image frames which are temporallyadjacent to each other.

(10) Preferably, the image processing apparatus further includes arecording unit recording, as a movie image file, a movie beingcompleted.

(11) Another image processing apparatus according to the presentinvention executes preprocessing to smoothly perform a movie display ofimage data composed of a plurality of still image frames obtained bystill-image photographing, including an image inputting unit, a frameinterpolating unit, a superimposing unit, and a smoothing unit.

The image inputting unit imports the plurality of the still imageframes.

The frame interpolating unit applies frame interpolation to theplurality of the still image frames in time-axis direction in order ofphotographing, to generate an interpolated image frame.

The superimposing unit performs moving addition of either one of “thestill image frame and the interpolated image frame” and “the pluralityof the interpolated image frames with each other” in the time-axisdirection, to generate a superimposed image frame.

The smoothing unit increases smoothness of the movie display bysuppressing more high spatial frequency components in the superimposedimage frame as a display rate of the movie display becomes lower.

(12) Another image processing apparatus according to the presentinvention executes preprocessing to smoothly perform a movie display ofimage data composed of a plurality of still image frames obtained bystill-image photographing, including an image inputting unit, a frameinterpolating unit, a superimposing unit, and a smoothing unit.

The image inputting unit imports the plurality of the still imageframes.

The frame interpolating unit applies frame interpolation to theplurality of the still image frames in time-axis direction in order ofphotographing, to generate an interpolated image frame.

The superimposing unit performs moving addition of either one of “thestill image frame and the interpolated image frame” and “the pluralityof the interpolated image frames with each other” in the time-axisdirection, to generate a superimposed image frame.

The smoothing unit increases smoothness of the movie display bysuppressing more high spatial frequency components in the superimposedimage frame as display zoom magnification of the movie display becomeshigher.

(13) A computer-executable image processing program according to thepresent invention allows a computer to function as the image inputtingunit, the frame interpolating unit, the superimposing unit, and thesmoothing unit.

(14) An image processing method according to the present inventionexecutes preprocessing to smoothly perform a movie display of image datacomposed of a plurality of still image frames obtained by still-imagephotographing, including the following steps.

(Step 1) The step importing the plurality of the still image frames.

(Step 2) The step applying frame interpolation to the plurality of thestill image frames in time-axis direction in order of photographing, togenerate an interpolated image frame.

(Step 3) The step performing moving addition of the still image frameand the interpolated image frame in the time-axis direction, to generatesuperimposed images frame sequentially.

(Step 4) The step increasing smoothness of the movie display bysuppressing more high spatial frequency components in the superimposedimage frame as alienation of motion between the still image framesbecomes larger.

BRIEF DESCRIPTION OF THE DRAWINGS

The nature, principle, and utility of the invention will become moreapparent from the following detailed description when read inconjunction with the accompanying drawings in which like parts aredesignated by identical reference numbers, in which:

FIG. 1 is a block diagram showing an electronic camera 100 in theembodiment;

FIG. 2 is an operation flowchart of a still image movie mode of theelectronic camera 100;

FIG. 3 is an operation flowchart of a movie processing mode;

FIGS. 4A and 4B are views explaining detection of motion vectors and agenerating operation of interpolated image frames; and

FIG. 5 is a view explaining a generating operation of superimposed imageframes.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, a preferred embodiment of the present invention will beexplained in detail based on the drawings.

(Structural Explanation of this Embodiment)

FIG. 1 is a block diagram showing an electronic camera 100 (including animage processing apparatus) according to the present embodiment.

As shown in FIG. 1, a lens 2 is attached to the electronic camera 100.Passing luminous flux of this lens 2 forms an image of a subject on animage sensor 1 via an aperture 13. This image sensor 1 is driven by adrive pulse from a driving unit 3 to thereby output image data of astill image frame.

This image data is inputted into a first processing unit 4 and subjectedto signal processing such as A/D conversion, white balance, gammaconversion, and electronic zoom.

Image data outputted from the first processing unit 4 is subjected toimage compression processing in a compression decoding unit 5 to beconverted into compressed data. This compressed data is combined withattendant information such as storage time information of the imagesensor 1 and a photographing time, and recorded successively as a stillimage file on a recording medium 6.

The still image file recorded on the recording medium 6 is decoded bythe compression decoding unit 5, and given to a second processing unit11 as image data of the still image frame. Incidentally, it is alsopossible that the second processing unit 11 directly receives the imagedata which is processed in the first processing unit 4.

In this second processing unit 11, image processing for reproduction isperformed according to an operation mode. When it is set to a stillimage movie mode, for example, the second processing unit 11 appliesimage processing of movie processing mode, which will be describedlater, to a image data group of the still image frames, to generatesuperimposed image frames for displaying the movie. These superimposedimage frames are reproduced and displayed on a display unit 12 as themovie. Further, these superimposed image frames are subjected to theimage compression in the compression decoding unit 5, and then recordedon the recording medium 6 as a movie file.

In addition, a control unit 10 and an operation member 8 are provided tothe electronic camera 100 so as to perform setting operation, systemcontrol and the like of the electronic camera 100.

Incidentally, the display unit 12 and the recording medium 6 may beprovided separately as external devices of the electronic camera 100. Inthis case, the display unit 12 and the recording medium 6 have signalconnection with the electronic camera 100 by cable splicing and radiointerfaces.

(Explanation of Imaging Operation of Electronic Camera 100)

FIG. 2 is an operation flowchart of the still image movie mode of theelectronic camera 100.

Hereinafter, the operation of the still image movie mode will beexplained according to step numbers shown in FIG. 2.

Step S1: When the control unit 10 receives the operation to startphotographing from the operation member 8 (release button), it instructsthe driving unit 3 to perform continuous photographing.

Step S2: The driving unit 3 performs shutter control by giving apredetermined charge storage time to the image sensor 1 in order toobtain a fine still image which has less blurring and can be appreciatedas the still image, to generate a still image frame. The shutter controlin this case may be either electronic shutter control or mechanicalshutter control. The driving unit 3 repeats this generating operation ofthe still image frame at such photographing intervals that a processingbuffer in the first processing unit 4 does not overflow.

These still image frames are processed by the first processing unit 4and the compression decoding unit 5, and recorded in order on therecording medium 6 as the still image files. The compressed data of thestill image frames, the charge storage times, the photographing timesand the like are stored in these still image files.

Step S3: The first processing unit 4 performs inter-frame differencearithmetic between the still image frames which are continuouslyphotographed, and detects an image motion amount between the still imageframes based on the result of the inter-frame difference arithmetic.

Step S4: The control unit 10 shortens the setting of the charge storagetime of the still image frame to be captured next as the image motionamount obtained from the first processing unit 4 becomes larger, inorder to prevent blurring of the subject. This processing allows thefine still image frame having less blurring to be obtained.

Incidentally, when it is possible for the image sensor 1 to control thestorage time for each pixel, the aforesaid processing may be appliedonly to the pixel corresponding to a partial area on an imaging surfacein which the motion is detected.

In this case, a signal level of the motion area is smaller as comparedto the area in which the motion is not detected, and hence levelcorrection processing is applied to data of the motion area so as tomake a correction thereto.

Step S5: The control unit 10 returns its operation to the step S2 andrepeats the continuous photographing of the still image frame, until itreceives the operation to complete the continuous photographing from theoperation member 8 (release button).

Meanwhile, when receiving the operation to complete the continuousphotographing, the control unit 10 completes the operation of the stillimage movie mode and moves its operation to the movie processing mode(described later) which generates movie image frame data with smoothmotion from still image frame data obtained as above.

(Operation Explanation of Movie Processing Mode)

FIG. 3 is an operation flowchart of the movie processing mode.

Hereinafter, the operation in the movie processing mode which is mainlyperformed by the second processing unit 11 will be explained accordingto step numbers shown in FIG. 3.

Step S11: The user can appropriately set a display frame rate of moviedisplay of the display unit 12 in advance by using a custom settingfunction of the electronic camera 100. The second processing unit 11gets the information on the setting of the display frame rate via thecontrol unit 10.

When the display unit 12 is provided separately as the external deviceof the electronic camera 100, the control unit 10 gets this informationthrough a connecting cable or the like to the display unit 12.

Step S12: The control unit 10 reads a still image file group (group of aseries of the files photographed in the still image movie mode) in therecording medium 6 successively, and gives them to the compressiondecoding unit 5. The compression decoding unit 5 decodes the still imagefile group successively, and outputs a plurality of the still imageframes to the second processing unit 11 in order.

The second processing unit 11 reads the photographing time of each stillimage frame from the file, to thereby get the information on aphotographing interval between the still image frames.

Step S13: The user can change display zoom magnification at the time ofdisplaying the movie on the display unit 12, by operating the operationmember 8 of the electronic camera 100. The second processing unit 11gets the information on the setting of the current display zoommagnification from the control unit 10.

Incidentally, when the display unit 12 is provided separately as theexternal device of the electronic camera 100, the control unit 10 canget this information through the connecting cable or the like to thedisplay unit 12.

The second processing unit 11 applies zoom processing (trimming andresolution conversion) to the still image frames, according to thedisplay zoom magnification.

Step S14: The second processing unit 11 performs block matching betweentemporally adjacent still image frames (hereinafter referred to as thepreceding frame and the succeeding frame). In this case, the precedingframe is broken down into a plurality of blocks, and the area whosepattern is close to each block is searched from the succeeding block.FIG. 4A is a view showing an example of motion vectors for therespective blocks, which are found from this search.

Incidentally, thus-found motion vectors may be smoothed between theadjacent blocks to increase space continuity of the motion vectors.Moreover, the search may be performed near new endpoints of the motionvectors after the smoothing, so as to improve the accuracy of the motionvectors.

Further, as to the block whose motion vector changes quitediscontinuously, it is preferable to further break down the block togenerate new blocks, and find motion vectors for the respective blocks.

Step S15: Based on the information obtained thus far, the secondprocessing unit 11 comprehensively evaluates alienation of the motion(awkwardness) AW between the still image frames, respectively, accordingto the following rules:

-   -   (1) the alienation is made larger as the charge storage time Tc        becomes shorter;    -   (2) the alienation is made larger as the photographing interval        Ti becomes longer;    -   (3) the alienation is made larger as the size of the maximum        motion vector Mmax in the image area becomes larger;    -   (4) the alienation is made larger as the display frame rate DF        becomes lower; and    -   (5) the alienation is made larger as the display zoom        magnification DZ becomes higher.

For example, the alienation AW may be found by using the followingevaluation equation.AW=w1(1/Tc)+w2×Ti+w×|Mmax|+w4(1/DF)+w 5×DZ  (Equation 1)

Incidentally, w1 to w5 are coefficients showing the degree of effectgiven to the awkwardness of the motion by respective factors of theabove (1) to (5) at the time of displaying the movie, and may bedetermined from an experiment and the like subjectively evaluating theawkwardness of the movie display.

Step S16: The second processing unit 11 sets the number of interpolatedimage frames and the number of added frames to increase as thealienation AW between the still image frames becomes larger. Inconcrete, the number of the interpolated image frames and the number ofthe added frames may be increased proportionally or monotonouslyaccording to the increase of the alienation AW. Alternatively, thenumber of the interpolated image frames and the number of the addedframes may be increased in stages according to the increase of thealienation AW.

Step S17: The second processing unit 11 internally divides the localmotion vector obtained by the block matching by the number of theinterpolated image frames set in the step S16, and finds internallydividing positions (P1 and P2 shown in FIG. 4B) for each block.

FIG. 4B shows in three dimensions the relationship between the twoframes of a still image frame K and a still image frame N which issucceedingly adjacent thereto in terms of time, and a motion vector V ofa local area F in the still image frame K.

In this case, it is supposed that the aforementioned number of theinterpolated image frames is two and interpolated image frames L and Mare generated. In this example, the area F in the still image frame Kmoves to an area FN in the still image frame N which succeeds thereto interms of time. It is the motion vector V that shows how the area Fmoves.

The interpolated image frames (Land M) are generated between the stillimage frame K and the still image frame N at equal time intervals.

Therefore, the positions of the local area F in the interpolated imageframes L and M are intersections P1 and P2 between the motion vector Vand the interpolated image frames L and M, respectively, as shown inFIG. 4B. Namely, P1 and P2 correspond to the internally dividingpositions which are the motion vector V being internally dividedaccording to the number of the interpolated image frames.

The second processing unit 11 extracts the local areas corresponding tostart/end points F and FN of the motion vector V from thepreceding/succeeding still image frames K and N, respectively, movesthem to the internally dividing positions (P1 and P2, for example), andperforms weighted addition according to internally dividing ratios(refer to FIG. 4B).

For example, in the example shown in FIG. 4B, the image data of thelocal area at the internally dividing position P1 is found by moving thelocal area F in the still image frame K and the local area FN in thestill image frame N to the internally dividing position P1 in theinterpolated image frame L, and adding the image data in both of thelocal areas at weighted ratios of 2/3 times and 1/3 times whichcorrespond to the internally dividing ratios, respectively. Similarly,the image data of the local area at the internally dividing position P2is found by moving the local area F in the still image frame K and thelocal area FN in the still image frame N to the internally dividingposition P2 in the interpolated image frame M, and adding the image datain both of the local areas at weighted ratios of 1/3 times and 2/3 timeswhich correspond to the internally dividing ratios, respectively.

By performing this weighted addition over the entire frame, it ispossible to generate the interpolated image frames by which the motionbetween the preceding/succeeding still image frames is artificiallydivided.

When it is so structured that thus-generated interpolated image framesare recorded on the recording medium 6 together with the still imageframes, superimposed image frames, which will be described later, can begenerated without performing interpolated image frame generatingprocessing, even when the alienation AW changes by changing the settingof the display frame rate, for example.

Step S18: The second processing unit 111 selects display frames to bethe basis of the display at equal intervals among the still image frameswhich are obtained by the photographing and the interpolated imageframes which are generated in the step S17, according to the informationon the display frame rate obtained in the step S11.

Display frames D shown in FIG. 5 are thus-selected display frames to bethe basis of the display.

Step S19: The second processing unit 111 extracts the still image framesand/or the interpolated image frames which are temporallypreceding/succeeding to the selected display frames to be the basis ofthe display by the number of the added frames found in the step S16, andperforms moving-addition in the time-axis direction, to generate thesuperimposed image frames.

Incidentally, in generating the superimposed image frames, the frameswhich are moving-added to the display frames to be the basis of thedisplay are not limited to the frames which are temporallypreceding/succeeding to the display frames to be the basis of thedisplay. Only the temporally preceding frames or only the temporallysucceeding frames may be used.

Additionally, it is preferable that the weight ratio of themoving-addition is made larger in the display frame to be the basis ofthe display, and made smaller with distance from the display frame to bethe basis of the display.

For example, in the example of FIG. 4B, when the number of the addedframes is three and the display frame to be the basis of the display isM, the image data of the frames L, M and N are added at the weightedratios of 1/4, 1/2 and 1/4, respectively, to obtain the superimposedimage frame. Here, the number of the added frames is set larger as thealienation AW becomes larger in the step S16.

At this time, it is preferable that a linkage of the motion isestablished between the superimposed image frames by overlapping theadded frames between the preceding/succeeding superimposed image frames.

For example, in the example of FIG. 4B, when the number of the addedframes is five and the still image frames K and N are selected as theframes to be the basis of the display, the interpolated image frames Land M are used as the succeeding frames to be added to the frame K, andthe interpolated image frames L and M are used as the preceding framesto be added to the frame N. Thus, the common frames (the interpolatedimage frames L and M in this case) are used in generating thesuperimposed image frames which are temporally adjacent to each other.

This means that, when displaying the superimposed image frames, theareas of the moving subject image are overlapped between thesuperimposed image frames which are temporally adjacent to each other.The larger the amount of the overlap, the smoother the motion appears.

In other words, it is preferable that the amount of the aforesaidoverlap (the number of the frames to be used repeatedly in generatingthe superimposed image frames which are temporally adjacent to eachother) is made larger as the alienation AW becomes larger. The amount ofthe overlap (the number of the frames) is set larger as the alienationAW becomes larger in the step S16.

The superimposed image frames C shown in FIG. 5 are thus-generatedsuperimposed image frames.

The example of FIG. 5 shows how the superimposed image frames C aregenerated by adding the image data of the two frames before and aftereach of the display frames D selected as the basis of the display.

Step S20: The second processing unit 11 determines whether the size ofthe maximum motion vector |Mmax| in the image area is larger than athreshold value α or not. The threshold value α is the threshold valuedeciding whether or not dynamic expression of a moving line is added toone frame of the movie, and is decided from the viewpoints ofnaturalness of the movie expression, intension of the expression, andthe like.

When the size |Mmax| is larger than the threshold value α, the secondprocessing unit 111 moves its operation to a step S21.

Meanwhile, when the size |Mmax| is equal to or smaller than thethreshold value α, the second processing unit 11 moves its operation toa step S22.

Step S21: The second processing unit 111 applies a smoothing filteraccording to the size and direction of the motion vectors over thesuperimposed image frame. This filter processing in the superimposedimage frame allows the moving line expression according to the motionvectors to be added to the superimposed image frame.

It should be noted that, instead of the smoothing filter, the smoothingmay be performed by removing high frequency components at the stage ofapplying compression coding to the superimposed image frame. In the caseof the compression in the JPEG system, for example, the high frequencycomponents can be removed by uniformly substituting high-frequency DCTcoefficients with zero, or by increasing a high-frequency quantizationstep.

In the above example, the case of performing the filter processing inthe superimposed image frame when the size of the maximum motion vector|Mmax| in the image area is larger than the predetermined thresholdvalue α is explained. However, filter processing characteristics in thesuperimposed image frame may be changed continuously according to thesize of the maximum motion vector |Mmax|.

In this case as well, the filter processing characteristics are changedso that the lower frequency is cut off according to high-frequencycutoff characteristics of the filter in the superimposed image frame asthe size of the maximum motion vector |Mmax| becomes larger.

Step S22: The second processing unit 11 displays the movie on thedisplay unit 12 by storing a group of the complete superimposed imageframes in a not-shown display buffer and the like.

Step S23: The second processing unit 111 gives the group of the completesuperimposed image frames to the compression decoding unit 5. Thecompression decoding unit 5 applies the compression coding to the groupof the superimposed image frames, generates the movie file, and recordsit on the recording medium 6.

By the processing described thus far, high spatial frequency componentsin the superimposed image frame are suppressed as the alienation of themotion between the still image frames becomes larger. As a result ofthis, it is possible to generate the superimposed image frames (moviefile) by which the smooth movie display is possible.

Incidentally, in this embodiment, the case of generating thesuperimposed image frame by performing the moving addition of the stillimage frames and the interpolated image frames, mainly, in the time-axisdirection is explained. However, the superimposed image frame may begenerated by adding the plural interpolated image frames with each otherin the time-axis direction.

(Effects and the Like of this Embodiment)

Next, the effects of this embodiment will be explained.

(1) First, the image processing apparatus of this embodiment takes inthe plural still image frames which are obtained by photographing thestill images. Next, the image processing apparatus applies interpolationbetween the still image frames in the time-axis direction, to generatethe interpolated image frame. The image processing apparatus performsthe moving addition of “the still image frame and the interpolated imageframe” or “the plural interpolated image frames with each other” in thetime-axis direction, to generate the superimposed image frame.

In each of the superimposed image frames, the images (motion) whosephotographing intervals are artificially shortened by interpolationprocessing are superimposed. By this superimposition, artificial imageflow is added to the superimposed image frames, and the alienation ofthe motion is reduced. As a result of this, smooth reproduction of themovie is made possible by using these superimposed image frames toreproduce the movie.

Further, according to this embodiment, the alienation of the motion isfound between the still image frames, and the superimposed image framesare made smoother as the alienation becomes larger. This processingallows the movie to be reproduced further smoother.

(2) It should be noted that, when the charge storage time at the time ofphotographing the still image frame becomes longer, the image flow ofthe moving subject becomes larger. The natural image flow of the stillimage frames reduces the awkwardness of the motion. On the contrary,when the charge storage time of the still image frame becomes shorter,the image flow becomes smaller to thereby cause big skips in the motionand the awkwardness of the motion.

Therefore, according to this embodiment, the high spatial frequencycomponents in the superimposed image frame are suppressed as the chargestorage time at the time of photographing the still image frame becomesshorter. By suppressing the high spatial frequency components, theartificial image flow of the superimposed image frames becomes larger,which makes it possible to improve the awkwardness of the motionaccompanying the shortened charge storage time.

(3) Moreover, when the photographing interval becomes longer, thepositions of the moving subject are separated between the frames, tothereby cause the awkwardness of the motion. Therefore, according tothis embodiment, the high spatial frequency components in thesuperimposed image frame are suppressed as the photographing interval ofthe still image frame becomes longer. By suppressing the high spatialfrequency components, the artificial image flow of the superimposedimage frames becomes larger, which makes it possible to improve theawkwardness of the motion accompanying the enlarged photographinginterval.

(4) Meanwhile, when the motion of the subject is fast, the positions ofthe moving subject are separated between the frames, to thereby causethe awkwardness of the motion. Therefore, according to this embodiment,the high spatial frequency components in the superimposed image frameare suppressed as a moving amount (motion amount) of the image becomeslarger between the still image frames. By suppressing the high spatialfrequency components, the artificial image flow of the superimposedimage frames becomes larger, which makes it possible to improve theawkwardness of the motion accompanying the fast movement of the subject.

(5) Incidentally, when the subject in the image area is small and themotion is detected in only one place, there is the possibility that thefast movement of the subject cannot be detected. Therefore, according tothis embodiment, the motion is found from a plurality of the places inthe image area, and the maximum motion amount is detected among these.This processing can improve the awkwardness of the motion accompanyingthe fast movement of the subject without fail.

(6) Incidentally, as the method of suppressing the high spatialfrequency components in the superimposed image frame, it is preferableto increase the number of the frames of the interpolated image frames.Then, the image flow which appears artificially in the superimposedimage frames can be made smooth.

(7) Additionally, as the method of suppressing the high spatialfrequency components in the superimposed image frame, it is alsopreferable to increase the number of the frames to be moving added.Then, the image flow which appears artificially in the superimposedimage frames can be made larger.

(8) Further, according to this embodiment, it is preferable to commonlyuse the still image frame or the interpolated image frame whengenerating the superimposed image frames which are temporally adjacentto each other. This processing gives an afterimage between thesuperimposed image frames, to thereby increase the smoothness of themovie display.

(9) Additionally, according to this embodiment, it is preferable toincrease the number of the still image frame or the interpolated imageframe to be used commonly when generating the superimposed image frameswhich are temporally adjacent to each other. Then, it is possible tosuppress the high spatial frequency components in the superimposed imageframe and to increase the smoothness of the movie display.

(10) Furthermore, according to this embodiment, it is preferable torecord the complete movie as the movie image file.

(11) Incidentally, the low display frame rate of the movie display alsocauses the skips in the motion of the subject and the awkwardness of themotion. Therefore, according to this embodiment, the high spatialfrequency components in the superimposed image frame are suppressed asthe display frame rate of the movie display becomes lower. Bysuppressing the high spatial frequency components, the artificial imageflow of the superimposed image frames becomes larger, which makes itpossible to improve the awkwardness of the motion accompanying theinsufficient display frame rate.

(12) Moreover, as the display zoom magnification of the movie displaybecomes higher, a motion width of the subject increases and theawkwardness is caused in the motion. Therefore, according to thisembodiment, the high spatial frequency components in the superimposedimage frame are suppressed as the display zoom magnification of themovie display becomes higher. By suppressing the high spatial frequencycomponents, the artificial image flow of the superimposed image framesbecomes larger, which makes it possible to improve the awkwardness ofthe motion accompanying the change in the magnification of the display.

(Supplementary Items of this Embodiment)

Incidentally, according to this embodiment, the still image frames aretemporarily recorded and then the superimposed image frames (movie file)are generated by postprocessing. However, this embodiment is not limitedto the above. For example, the still image frames outputted from thefirst processing unit 4 may be given to the second processing unit 11 tothereby generate the superimposed image frames (movie file) directly.

Further, according to this embodiment, the explanation about theelectronic camera 100 has been given. However, this embodiment is notlimited to the above. For example, the processing operation shown inFIG. 3 may be programmed and the computer may execute the program.

Furthermore, by using the method of the image processing of thisembodiment in a server machine on the Internet and the like, it ispossible to provide image processing service such as an electronicalbum.

The invention is not limited to the above embodiments and variousmodifications may be made without departing from the spirit and scope ofthe invention. Any improvement may be made in part or all of thecomponents.

1. An image processing apparatus executing preprocessing to smoothlyperform a movie display of image data composed of a plurality of stillimage frames obtained by still-image photographing, comprising: an imageinputting unit importing the plurality of said still image frames; aframe interpolating unit applying frame interpolation to the pluralityof said still image frames in time-axis direction in order ofphotographing, to generate an interpolated image frame; a superimposingunit performing moving addition of either one of “said still image frameand said interpolated image frame” and “a plurality of said interpolatedimage frames with each other” in said time-axis direction, to generate asuperimposed image frame; and a smoothing unit increasing smoothness ofsaid movie display by suppressing more high spatial frequency componentsin said superimposed image frame as alienation of motion between saidstill image frames becomes larger.
 2. The image processing apparatusaccording to claim 1, wherein said smoothing unit increases thesmoothness of said movie display by suppressing more of the high spatialfrequency components in said superimposed image frame as a chargestorage time at the time of photographing said still image frame becomesshorter.
 3. The image processing apparatus according to claim 1, whereinsaid smoothing unit increases the smoothness of said movie display bysuppressing more of the high spatial frequency components in saidsuperimposed image frame as a photographing interval of said still imageframe becomes longer.
 4. The image processing apparatus according toclaim 1, wherein said smoothing unit increases the smoothness of saidmovie display by suppressing more of the high spatial frequencycomponents in said superimposed image frame as a moving amount (motionamount) of the image between said still image frames becomes lager. 5.The image processing apparatus according to claim 4, wherein saidsmoothing unit suppresses more of the high spatial frequency componentsin said superimposed image frame as a maximum motion amount among themotion found from a plurality of places in an image area becomes larger.6. The image processing apparatus according to claim 1, wherein saidsmoothing unit increases the smoothness of said movie display byincreasing a number of the frames of said interpolated image frameswhich said frame interpolating unit generated, and by suppressing thehigh spatial frequency components in said superimposed image frame. 7.The image processing apparatus according to claim 1, wherein saidsmoothing unit increases the smoothness of said movie display byincreasing a number of frames to be performed moving addition, and bysuppressing the high spatial frequency components in said superimposedimage frame.
 8. The image processing apparatus according to claim 1,wherein said smoothing unit increases the smoothness of said moviedisplay by commonly using either one of said still image frame and saidinterpolated image frame when generating said superimposed image frameswhich are temporally adjacent to each other.
 9. The image processingapparatus according to claim 8, wherein said smoothing unit increasesthe smoothness of the movie display by increasing a number of either oneof the still image frame and the interpolated image frame to be usedcommonly, and by suppressing the high spatial frequency components inthe superimposed image frame, when generating the superimposed imageframes which are temporally adjacent to each other.
 10. The imageprocessing apparatus according to claim 1, further comprising arecording unit recording, as a movie image file, a movie beingcompleted.
 11. An image processing apparatus executing preprocessing tosmoothly perform a movie display of image data composed of a pluralityof still image frames obtained by still-image photographing, comprising:an image inputting unit importing the plurality of said still imageframes; a frame interpolating unit applying frame interpolation to theplurality of said still image frames in time-axis direction in order ofphotographing, to generate an interpolated image frame; a superimposingunit performing moving addition of either one of “said still image frameand said interpolated image frame” and “the plurality of saidinterpolated image frames with each other” in said time-axis direction,to generate a superimposed image frame; and a smoothing unit increasingsmoothness of said movie display by suppressing more high spatialfrequency components in said superimposed image frame as a display rateof said movie display becomes lower.
 12. An image processing apparatusexecuting preprocessing to smoothly perform a movie display of imagedata composed of a plurality of still image frames obtained bystill-image photographing, comprising: an image inputting unit importingthe plurality of said still image frames; a frame interpolating unitapplying frame interpolation to the plurality of said still image framesin time-axis direction in order of photographing, to generate aninterpolated image frame; a superimposing unit performing movingaddition of either one of “said still image frame and said interpolatedimage frame” and “the plurality of said interpolated image frames witheach other” in said time-axis direction, to generate a superimposedimage frame; and a smoothing unit increasing smoothness of said moviedisplay by suppressing more high spatial frequency components in saidsuperimposed image frame as display zoom magnification of said moviedisplay becomes higher.
 13. A computer-executable image processingprogram allowing a computer to function as said image inputting unit,said frame interpolating unit, said superimposing unit, and saidsmoothing unit according to claim
 1. 14. An image processing methodexecuting preprocessing to smoothly perform a movie display of imagedata composed of a plurality of still image frames obtained bystill-image photographing, comprising the steps of: importing theplurality of said still image frames; applying frame interpolation tothe plurality of said still image frames in time-axis direction in orderof photographing, to generate an interpolated image frame; performingmoving addition of said still image frame and said interpolated imageframe in said time-axis direction, to generate superimposed image framessequentially; and increasing smoothness of said movie display bysuppressing more high spatial frequency components in said superimposedimage frame as alienation of motion between said still image framesbecomes larger.
 15. The image processing apparatus according to claim 7,wherein said smoothing unit further increases the smoothness of saidmovie display by applying smoothing filter over said superimposed imagethat has been performed moving addition, and changes, as said alienationof motion becomes larger, a processing characteristic of a smoothingfilter so that either one of a cutting off of lower frequencies in ahigh-frequency cutoff characteristic of said smoothing filter in saidsuperimposed image frame, and an increasing of a high spatial frequencycutoff amount of said smoothing filter in said superimposed image frame,is performed.
 16. The image processing apparatus according to claim 15,wherein said smoothing filter is a filter that performs smoothing on adirection of said motion.
 17. The image processing apparatus accordingto claim 10, wherein said smoothing unit includes a compression unitthat compresses data of said movie being completed and supplies acompressed image data to said recording unit, in which the compressionunit, according to said alienation of motion, generates said compressedimage data by performing a high-frequency cutoff processing in thesuperimposed image frame when compression-encoding said superimposedimage frame.
 18. The image processing apparatus according to claim 11,wherein said smoothing unit further increases the smoothness of saidmovie display by applying smoothing filter that performs smoothing on adirection of a motion between frames over said superimposed image thathas been performed moving addition, and changes, as said display framerate becomes lower, a processing characteristic of a smoothing filter sothat either one of a cutting off of lower frequencies in ahigh-frequency cutoff characteristic of said smoothing filter in saidsuperimposed image frame, and an increasing of a high spatial frequencycutoff amount of said smoothing filter in said superimposed image frame,is performed.